Method and apparatus for generating and interfacing with a haptic virtual reality environment

ABSTRACT

A method for generating a haptic interactive representation including the steps of defining a haptic interaction space and building a hierarchical construct, for use within the haptic interaction space, using a plurality of underlying constructs. In one embodiment the method includes determining the forces to be applied to a user by generating a haptic interactive, sensing a position of a user in real space, determining a haptic interface location in the haptic interaction space in response to the position of the user in real space and determining whether the virtual object collides with the haptic interface location. The invention also relates to a method for interacting with a haptic interactive representation. The method includes creating a haptic process having the haptic interactive representation, associating a callback with the haptic process, connecting to a second process which interacts with the haptic process and associating the callback which is associated with the haptic process with the second process. The method further includes requesting the callback by the second process, and returning, by the haptic process through the callback to the second process, at least one parameter associated with the haptic interactive representation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of co-pending U.S. patentapplication Ser. No. 11/968,474, filed Jan. 2, 2008, which is acontinuation application of U.S. patent application Ser. No. 09/574,131,filed May 18, 2000, and issued on Jan. 15, 2008, as U.S. Pat. No.7,319,466, which is a continuation application of U.S. patentapplication Ser. No. 08/691,247, filed Aug. 2, 1996, and issued on Jul.4, 2000, as U.S. Pat. No. 6,084,587, the entire contents of which areincorporated by reference herein.

FIELD OF THE INVENTION

The invention relates generally to methods for defining a virtualreality environment and more specifically to methods for hapticallycreating and manipulating objects within a virtual reality environment.

BACKGROUND OF THE INVENTION

Virtual Reality (VR) is an artificial environment constructed by acomputer which permits the user to interact with that environment as ifthe user were actually immersed in the environment. Early VR devicespermitted the user to see three dimensional (3D) depictions of anartificial environment and to move within that environment. Thus a VRflight simulator incorporating such a device would allow a user to see a3D view of the ground, which changed both as the virtual aircraft passedover the virtual ground and as the user's eyes looked in differentdirections. What the user saw in such a simulator is what a pilot wouldsee when actually flying an aircraft.

The reality of the VR environment was enhanced by the ability of a userto manipulate virtual objects within the virtual environment using handmotions and gestures. Special gloves and devices were developed whichpermitted the user to interact with the virtual objects within thevirtual environment. In such a system the user typically saw an image ofhis or her hand within the virtual environment and was able to determinewhere in the virtual environment the user's hand was relative to thevirtual object to be manipulated. Moving the glove or device resulted ina corresponding movement of the hand image in the virtual environment.Thus a user wearing the special gloves or using the special device wouldcause virtual objects to move, simply by moving the glove in such a waythat the virtual object is touched by the image of the hand in thevirtual environment.

The addition of force generators to the gloves or devices furtherenhanced the reality of VR by providing to the user with a tactualresponse to interacting with virtual objects within the virtualenvironment. For example a user moving such a force generation enhanceddevice in a direction such that the image of the device moved toward avirtual wall in the virtual environment, would experience a stoppingforce when the image of the device in the virtual environment collidedwith the virtual wall. Such tactual sensation creating devices, orhaptic devices, thus provided a stimulus to another of the user'ssenses.

Although the early VR devices were oriented to providing realisticgraphical interfaces, the progress in haptic VR devices has made itpossible to define a haptic VR environment which may be completelyindependent of the graphical VR environment. As such, haptic VRenvironments may now be constructed which respond to manipulation by theuser in the way that the early graphical VR environments responded tothe visual actions of a user.

This ability to define a haptic VR space independently of a graphical,or other space, provides a greater degree of flexibility in the designand creation of such VR environments. The present invention seeks tofurther add to this flexibility.

SUMMARY OF THE INVENTION

The invention relates to a method for generating a haptic interactiverepresentation. The method includes defining a haptic interaction spaceand building a hierarchical construct for use within the hapticinteraction space. In one embodiment the step of building thehierarchical construct for use within the haptic interaction spaceincludes the step of creating a hierarchical representation of an objectusing a plurality of underlying constructs. The hierarchicalrepresentation of the object may be created in the haptic interactionspace or outside of the haptic interaction space and then placed intothe haptic interaction space. The underlying constructs includegeometric representations, of the same or dissimilar form. Suchrepresentations include primitive objects, polygonal objects, algebraicobjects, constructed solid geometry objects, algebraic surfacerepresentations and volumetric object representations. Other underlyingconstructs include constructs associated with translation parameters,rotation parameters, scaling parameters, mass parameters, velocityparameters, acceleration parameters, angular velocity parameters,angular acceleration parameters, friction parameters, textureparameters, viscosity parameters, and compliance parameters.

In one embodiment, the invention includes a method for determiningforces to be applied to a user through a haptic interface. The methodincludes generating a haptic interactive representation including avirtual object in a haptic interaction space, sensing a position of auser in real space, determining a haptic interface location in thehaptic interaction space in response to the position of the user in realspace and determining whether the virtual object collides with thehaptic interface location. If the virtual object does not collide withthe haptic interface location, a force to be applied to the user in realspace in response to the haptic interface location is calculated. If thevirtual object collides with the haptic interface location, a proposedsurface contact point for the virtual object based on the hapticinterface location is determined and a force to be applied to the userin real space, in response to the proposed surface contact point, iscalculated.

In another embodiment the method includes determining forces to beapplied to a user through a haptic interface by generating a hapticinteractive representation including at least two virtual objects in ahaptic interaction space, sensing a position of a user in real space,determining a current haptic interface location in the hapticinteraction space in response to the position of the user in real spaceand determining whether each of the virtual objects collides with thehaptic interface location. If none of the virtual objects collide withthe haptic interface location, a force to be applied to the user in realspace in response to the haptic interface location is calculated. Ifonly one of the virtual objects collides with the haptic interfacelocation, a proposed surface contact point for that virtual objectcolliding with the haptic interface location is determined, and a forceto be applied to the user in real space, in response to the proposedsurface contact point, is calculated. If more than one of the virtualobjects collide with the haptic interface location, a proposed surfacecontact point for each of the virtual objects colliding with said hapticinterface location is determined to thereby define a plurality ofproposed surface contact points. A force to be applied to the user inreal space, in response to the plurality of proposed surface contactpoints, is then calculated.

The invention also relates to a method for interacting with a hapticprocess. The method includes the steps of creating a haptic processhaving the haptic interactive representation, associating a callbackwith the haptic process, connecting to a second process which interactswith the haptic process and associating the callback, which isassociated with the haptic process, with the second process. The methodfurther includes requesting the callback by the second process, andreturning, by the haptic process through the callback to the secondprocess, at least one parameter associated with the haptic interactiverepresentation.

In one embodiment the step of creating a haptic process includes thesteps of defining a haptic interaction space, and building ahierarchical construct for use within the haptic interaction space. Inanother embodiment the step of connecting to the second process includesthe steps of establishing a communications link between the hapticprocess and the second process, and permitting the haptic process andthe second process to communicate over the communications link.

In one embodiment the communications link is a computer network and thesecond process is active on a node different from the haptic process. Inone embodiment the second process is a graphics process and the at leastone parameter includes graphical data. In another embodiment the secondprocess is a second haptic interactive process and the at least oneparameter is comprises haptic data.

BRIEF DESCRIPTION OF THE DRAWINGS

This invention is pointed out with particularity in the appended claims.The above and further advantages of this invention may be betterunderstood by referring to the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 is a flowchart representation of an embodiment of a process forgenerating a haptic interactive representation;

FIG. 1A is a flowchart representation of an embodiment of a process forcreating a hierarchical representation of an object;

FIG. 2 is a pictorial view of an embodiment of a hierarchical constructfor use within a haptic interaction space;

FIG. 3 is a flowchart representation of an embodiment of a process fordetermining a feedback force to be applied to a user through a hapticinterface when the haptic interactive representation contains only onevirtual object;

FIG. 4 is a flowchart representation of an embodiment of a process fordetermining a feedback force to be applied to a user through a hapticinterface when the haptic interactive representation contains aplurality of virtual objects;

FIG. 5 is a flowchart representation of an embodiment of a process fordetermining a feedback force to be a applied to a user through a hapticinterface when more than one virtual object collides with the hapticinterface location;

FIG. 6 is a pictorial view of a surface of a virtual object and asurface contact point located on the surface;

FIG. 7 is a flowchart representation of an embodiment of a process fordetermining a friction force to be applied to a user through a hapticinterface;

FIG. 8 is a flowchart representation of an embodiment of a process fordetermining the initial proposed surface contact points for algebraicobjects contained in the haptic interactive representation;

FIG. 9 is a flowchart representation of an embodiment of a process fordetermining proposed surface contact points for algebraic objectscontained within the haptic interactive representation after initialcontact with the virtual objects;

FIG. 9A is a pictorial view of an algebraic surface and a hapticinterface point which has penetrated the algebraic surface;

FIG. 10A is a flowchart representation of an embodiment of a process fordetermining proposed surface contact points for polygonal objectscontained within the haptic interactive representation;

FIG. 10B is a flowchart representation of an embodiment of a process fordetermining a proposed surface contact point for a polygonal object whenthe haptic interface is not located solely within one facet of thepolygonal object;

FIG. 11 is a flowchart representation of another embodiment of a processfor determining a feedback force to be applied to a user through ahaptic interface;

FIG. 11A is a flowchart representation continuing the steps of theprocess of FIG. 11;

FIG. 12A is a pictorial view of a virtual object colliding with a hapticinterface;

FIG. 12B is a pictorial view of two virtual objects colliding with ahaptic interface;

FIG. 13 is a block diagram of an embodiment of multiple processescommunicating with a haptic process using callback mechanism;

FIG. 14 is a flow diagram of a method of establishing and using acallback to communicate between a haptic process and another process;

FIG. 15 is a flowchart representation of another embodiment of a processfor determining the reaction forces to be applied to a user in realspace through a haptic interface device;

FIG. 16 is a flow diagram of one embodiment of the invention; and

FIG. 17 is a flowchart representation of an embodiment of a process forgenerating a virtual reality environment.

Like reference characters in the respective drawn figures indicatecorresponding parts.

DETAILED DESCRIPTION OF THE INVENTION

In brief overview, the flowchart of FIG. 1 shows the steps performed byone embodiment of the method of the present invention for generating ahaptic interactive representation. As used herein, a “haptic interactiverepresentation” is defined as a computer generated virtual environmentwhich can be explored by a user through the sense of touch. First, (step10) a haptic rendering application defines a haptic interaction space.As used herein, “haptic rendering” is defined as the creation of avirtual environment with which a user can interact through the sense oftouch. “Haptic rendering application” refers to the computer applicationwhich generates the virtual environment and determines the forces to beapplied to a user through a haptic interface. The haptic renderingapplication generates representations of real world objects in thevirtual environment. As used herein, “haptic interaction space” isdefined as the region in the computer generated virtual environment withwhich the user can interact through the sense of touch. The hapticinteraction space defines the boundaries of the haptic interactiverepresentation with which the user can interact. In one embodiment, thehaptic interaction space is a two-dimensional region. In anotherembodiment, the haptic interaction space is a three-dimensional region.The user selects the desired dimensions for the haptic interactionspace.

After the user has defined the haptic interaction space, the user buildsa hierarchical construct for use within the haptic interaction space.The hierarchical construct provides the virtual environment forinteraction with the user. Hierarchical constructs are used to modelobjects that are made up of several parts which are linked together andwhich can interact with each other. FIG. 2 shows an example of ahierarchical construct 12 for use within a haptic interaction space. Thehierarchical construct 12 is a model of a virtual world 13 whichcontains three virtual objects 14, 16 and 18. The world 13 is ahierarchical construct because the world 13 is made up of parts (i.e.the three objects 14, 16, 18), the parts consist of subparts, and so on.The three objects 14, 16 and 18 are sub-constructs of the hierarchicalconstruct 12. Each of the sub-constructs 14, 16 and 18 are composed of aplurality of further underlying sub-constructs which are linked togetherto form a hierarchy. For example, OBJECT₁ 14 is a virtual representationof a body 20. The body 20 includes two arms 22, 24. The arms 22, 24 aresub-constructs or “children” of the body 20 which are linked together toform the body 20. As used herein, a “child” is a sub-construct that hasa position and orientation which is dependent upon the position andorientation of a construct higher up in the hierarchy. The arms 22, 24are children of the body 20 because the position and orientation of thearms 22, 24 are dependent upon the position of the body 20. The arm 22includes a hand 26, which includes two fingers 28, 30. The two fingers28, 30 are sub-constructs or children of the hand 26 and the hand 26 isa sub-construct or child of the arm 22.

Referring again to FIG. 1, to build the hierarchical construct, in step32 the user determines the real world objects that the user desires toexist in the haptic interaction space. Each of these desired objects aresub-constructs of the hierarchical construct representing the entirehaptic interactive representation. In step 34, the user creates ahierarchical representation of each of the desired objects. Next, instep 36 the user places the hierarchical representations of the desiredobjects into the haptic interaction space to create the hapticinteractive representation. In another embodiment, the user creates thehierarchical representation directly in the haptic interaction space.After the user creates the haptic interactive representation, in step 38the user may interact with the representations of the desired objectswithin the haptic interaction space.

In more detail and referring to FIG. 1A, a flowchart illustrates a moredetailed sequence of steps performed by a user in one embodiment of thepresent invention to create a hierarchical representation of each of theobjects the user desires to exist in the haptic interaction space. Instep 40, the user decomposes each hierarchical representation into aplurality of simpler sub-constructs. In the example illustrated in FIG.2, the hierarchical representation of the body 20 is decomposed intosimpler sub-constructs which include the arms 22, 24, the hand 26, andthe fingers 28, 30. After decomposing each hierarchical representationinto a plurality of sub-constructs, the user builds each of thesub-constructs using a plurality of underlying constructs. Theunderlying constructs include the attributes and properties that a usermay use to build the sub-constructs. The underlying constructs affecthow the object appears in the haptic interaction space. For example, theset of underlying constructs the user may select from to build thesub-constructs include geometric representations, translationparameters, rotation parameters, and scaling parameters. Similarly, theunderlying representations may also include mass parameters, velocityparameters, acceleration parameters, angular velocity parameters,angular acceleration parameters, friction parameters, textureparameters, viscosity parameters and compliance parameters.

In step 42, the user selects a base geometric representation for each ofthe sub-constructs. As used herein, a “geometric representation” isdefined as a geometric shape. The geometric representation for eachsub-construct may be a primitive object, a polygonal object, analgebraic object, a constructed solid geometry object, a volumetricobject representation or a representation utilizing Octreedecompositions of the haptic interaction space. Examples of primitiveobjects include lines, circles and curves. Polygonal objects are objectscreated by linking polygons. Algebraic objects are objects that aredefined by an algebraic equation. Volumetric objects are objects whichare formed by filling a region of the haptic interaction space with avolume and removing pieces of that volume to form a shape. An Octreeobject is one defined by successively dividing haptic spaces intooctants until a predetermined volume element, voxel, resolution isreached. At this resolution, each voxel is marked to indicate whetherthe volume element contains a portion of the object. In one embodiment,the geometric representations used to build the sub-constructs are ofdissimilar form. For example, one sub-construct in the hapticinteractive representation may utilize a primitive object as its basegeometric representation, while a second sub-construct utilizes analgebraic object as its base geometric representation.

After selecting a base geometric representation for each of thesub-constructs, in step 44 the user operates on the sub-constructs bytranslating, rotating and scaling the geometric representations. If theuser did not operate on the sub-constructs, each of the sub-constructswould be located at a default position within the haptic interactiverepresentation, and would have a default orientation and a default size.For example, in the hierarchical construct shown in FIG. 2, thesub-construct FINGER₁ 28 has a base geometric representation 46. In oneembodiment, the base geometric representation 46 is a sphere. The userthen associates a translation parameter 48, a rotation parameter 50 anda scaling parameter 52 with the sub-construct FINGER₁ 28. Associatingthe translation parameter 48, the rotation parameter 50 and the scalingparameter 52 with the sub-construct FINGER₁ 28 has the effect oftranslating the geometric representation to the desired position withinthe haptic interactive representation, rotating the geometricrepresentation to the desired orientation and scaling the geometricrepresentation until it more closely resembles a finger. To cause thegeometric representation to more closely resemble a finger in shape, theuser may scale one point of the sphere until the sphere is more oblongin shape.

In another embodiment, the user may associate a stiffness, compliance,and friction parameter with each sub-construct.

After the user translates, rotates and scales the geometricrepresentations, in step 54 the user may associate dynamic propertieswith each sub-construct. The dynamic properties that a user mayassociate with each sub-construct include velocity, acceleration,angular velocity and angular acceleration. Finally, in step 56, the userlinks the sub-constructs together to form the hierarchicalrepresentations of the desired objects.

Referring now to FIG. 3, a flowchart shows the steps performed by oneembodiment of the method of the present invention for determining theforces to be applied to a user through a haptic interface device. Instep 60, the haptic rendering application generates a haptic interactiverepresentation which includes only one representation of a real worldobject. As used herein, “virtual object” is defined as therepresentation of a real world object in the haptic interactiverepresentation. In one embodiment, the virtual object is a hierarchicalrepresentation of a real world object. The user may create thehierarchical representation of the object using underlying constructs asdescribed above. In the embodiment illustrated by the flowchart in FIG.3, the haptic interactive representation contains only one virtualobject. The haptic interactive representation may contain more than oneobject. An embodiment of a haptic interactive representation containingmore than one virtual object will be described in more detail below.

In step 62, sensors of the haptic interface system sense the position ofthe user in real space. As used herein, “real space” is defined as thereal world environment. In step 64, the haptic rendering applicationutilizes the information obtained by the sensors to determine the hapticinterface in haptic interaction space. The location of the hapticinterface describes the position of the user in the haptic interactionspace. In step 66, the haptic rendering application determines whetherthe virtual object collides with the haptic interface location. Avirtual object “collides” with the haptic interface location if thesurface of the virtual object is in contact with the haptic interfacelocation or if the virtual object encompasses the haptic interfacelocation. If the virtual object does not collide with the hapticinterface location, in step 68 the haptic rendering applicationcalculates the force to be applied to the user in real space in responseto the haptic interface location. If the virtual object collides withthe haptic interface location, in step 70 the haptic renderingapplication determines a proposed surface contact point (SCP) for thevirtual object. The surface contact point represents the location on thesurface of the virtual object at which the haptic interface would belocated if the haptic interface could be prevented from penetratingvirtual objects. The purpose of the surface contact point is to enablethe haptic rendering application to accurately determine the forces tobe applied to the user in real space. The methods used to determine theproposed surface contact point will be described in more detail below.

After the haptic rendering application determines the proposed SCP forthe virtual object, in step 72 the haptic rendering applicationcalculates the force to be applied to the user in real space based onthe location of the proposed SCP. After the haptic rendering applicationhas calculated the force to be applied to the user in either step 68 orstep 72, in step 74 this force may be generated and applied to a user inreal space through a haptic interface device. After the force has beenapplied to the user, the haptic rendering application repeats steps 62,64, 66, 68, 70, 72 and 74 for the duration of time that the userinteracts with the haptic interactive representation.

FIG. 4 shows a flowchart illustrating the steps performed by anotherembodiment of the method of the present invention for determining theforces to be applied to a user through a haptic interface device. In theembodiment illustrated by the flowchart of FIG. 4, the user'sinteractions with the haptic interactive representation are reduced tothose of a point interacting with three dimensional objects in a threedimensional virtual environment. In other embodiments, the user'sinteractions are not reduced to those of a point interacting with threedimensional virtual objects. In other embodiments, the haptic interfacemay be a set of points. In still other embodiments, the haptic interfacemay be a three-dimensional object.

In step 76, the haptic rendering application generates a hapticinteractive representation which includes at least two representationsof virtual objects. Once the haptic rendering application has generateda haptic interactive representation which includes at least two virtualobjects, in step 78 the haptic interface device senses the position ofthe user in real space. In step 80, the haptic rendering applicationuses the position of the user in real space to determine the location ofthe haptic interface in the haptic interaction space. When the userchanges position, the haptic interface device senses this change inposition and the haptic rendering application updates the location ofthe haptic interface in haptic interaction space to reflect the changeof the user's position in real space.

Once the haptic rendering application determines the location of thehaptic interface, in step 82 it determines whether any of the virtualobjects collide with the haptic interface location. A method fordetermining whether a virtual object collides with the haptic interfacelocation will be discussed in more detail below. If no virtual objectscollide with the haptic interface, in step 84 the haptic renderingapplication calculates the force to be applied to the user in real spacebased on the location of the haptic interface. If at least one of thevirtual objects collides with the haptic interface location, in step 86the haptic rendering application calculates a proposed surface contactpoint (SCP) for each virtual object colliding with the haptic interfacelocation. In step 88, the haptic rendering application determines ifmore than one virtual object collides with the haptic interfacelocation. If only one virtual object collides with the haptic interfacelocation, in step 90 the haptic rendering application calculates a forceto be applied to the user in real space in response to the proposed SCP.If more than one virtual object collides with the haptic interfacelocation, in step 92 the haptic rendering application calculates a forceto be applied to a user in real space in response to the set of proposedSCPs. The method for determining the SCP to use to calculate the forcewill be described in more detail below in the discussion of FIG. 5.

FIG. 5 shows a flowchart illustrating the steps performed by oneembodiment of the method of the present invention for determining aforce to be applied to a user through a haptic interface when more thanone virtual object collides with the haptic interface location. Asdiscussed above, when more than one virtual object collides with thehaptic interface location the haptic rendering application determines aproposed SCP for each virtual object colliding with the haptic interfacelocation. In step 94, the haptic rendering application determines if anyof the proposed SCPs is not located within any of the virtual objects inthe haptic interactive representation. If at least one of the proposedSCPs is not located within any of the virtual objects, in step 96 thehaptic rendering application calculates a force to be applied to theuser in response to that proposed SCP. If more than one proposed SCP isnot located within any of the virtual objects, the haptic renderingapplication will use the first SCP it determines not to be locatedwithin any of the virtual objects to calculate the force to be appliedto the user.

If all of the proposed SCPs are located within at least one virtualobject, in step 98 the haptic rendering application determines if morethan two virtual objects collide with the haptic interface location. Ifmore than two virtual objects collide with the haptic interface, in step100 the haptic rendering application discards the proposed SCPs andcalculates a force to be applied to the user in response to the currentSCP. If only two virtual objects collide with the haptic interfacelocation, in step 102 the haptic rendering application determines asurface contact point normal for both proposed surface contact points.As used herein, a “surface contact point normal” is the vector normal tothe surface of the virtual object at the location of that object'sproposed SCP. In step 104, the haptic rendering application calculatesthe cross product of the two surface contact point normals to determinea vector (V_(PERP)) perpendicular to the two surface contact pointnormals as illustrated by equation (1) below. In equation (1),SCP_(1NORMAL) and SCP_(2NORMAL) represent the surface contact pointnormals for the two virtual objects colliding with the haptic interfacelocation.

V _(PERP)=SCP_(1NORMAL)×SCP_(2NORMAL)  (1)

In step 106, the haptic rendering application determines the line whichpasses through the current SCP and which is parallel to the vectorV_(PERP). In step 108 the haptic rendering application projects thehaptic interface location onto the line determined in step 106 in orderto determine an updated SCP location. Finally in step 110, the hapticrendering application calculates a force to be applied to the user inreal space in response to the updated SCP location determined in step108.

FIGS. 6 and 7 illustrate the steps performed by one embodiment of ahaptic rendering application to calculate a friction force to be appliedto the user in real space. FIG. 6 shows a surface 112 of a portion of avirtual object 114. FIG. 6 shows the haptic interface point moving froma first haptic interface point location (P_(HI)) 116 to a second hapticinterface point location (P′_(HI)) 118. The point SCP_(CURRENT) 120represents the current location of the surface contact point. The hapticrendering application calculated the point SCP_(CURRENT) 120 in responseto the first haptic interface point location (P_(HI)) 116. The pointSCP_(PROPOSED) 122 represents the haptic rendering application'sproposed new surface contact point. The haptic rendering applicationcalculated the point SCP_(PROPOSED) 122 in response to the change inlocation of the haptic interface point from P_(HI) 116 to P′_(HI) 118.

FIG. 7 shows a flowchart of the steps performed by one embodiment of themethod of the present invention for determining the friction forces tobe applied to a user in the real world through a haptic interfacedevice. The method illustrated by the flowchart in FIG. 7 determinesboth the static and kinetic friction forces to be applied to the user inreal space. The method calculates a static friction force to be appliedto the user if the user has made contact with a virtual object, but hasnot provided enough tangential force to “break away” from the object.Once the user applies enough force to break away from the object, thehaptic rendering application determines a kinetic friction force to beapplied to the user. The kinetic friction force is applied in adirection to oppose the direction of motion of the user. All frictionforces are applied to the user in directions tangential to the surfaceof the virtual object being contacted.

In step 124, the haptic rendering application determines thedisplacement (ΔSCP) between the proposed surface contact pointSCP_(PROPOSED) 122 and the current surface contact point SCP_(CURRENT)120. Next, in step 126, the haptic rendering application determines thedisplacement (d) between the proposed surface contact pointSCP_(PROPOSED) 122 and the current haptic interface point P′_(HI) 118.In step 128, the haptic rendering application multiplies thedisplacement d by a coefficient of friction (k_(f)) to determine aproduct. The haptic rendering application then determines if the productof the displacement d and the coefficient of friction k_(f) is less thanthe displacement ΔSCP as illustrated by equation (2) below.

d·k _(f)<ΔSCP  (2)

If the product of the displacement d and the coefficient of frictionk_(f) is less than the displacement ΔSCP, the haptic renderingapplication proceeds to step 130 and places the proposed surface contactpoint SCP_(PROPOSED) 122 at the location of the current surface contactpoint SCP_(CURRENT) 120. Next, in step 132, the haptic renderingapplication calculates a friction force to be applied to the user inresponse to the current surface contact point SCP_(CURRENT) 120. Incalculating the friction force, the haptic rendering application usesthe coefficient of static friction associated with the location of thecurrent surface contact point SCP_(CURRENT) 120. If the product of thedisplacement d and the coefficient of friction k_(f) is greater than orequal to the displacement ΔSCP, the haptic rendering applicationproceeds to step 134 and updates the location of the proposed surfacecontact point SCP_(PROPOSED) 122 to an new proposed surface contactpoint (SCP′_(PROPOSED)) according to equation (3) below.

$\begin{matrix}{{{SCP}^{\prime}}_{PROPOSED} = {{SCP}_{CURRENT} + {\frac{( {\Delta \; {SCP}} )}{{\Delta \; {SCP}}} \cdot ( {{{\Delta \; {SCP}}} - {d \cdot k_{f}}} )}}} & (3)\end{matrix}$

In equation (3), SCP′_(PROPOSED) represents the updated location of theproposed surface contact point, SCP_(CURRENT) represents the location ofthe current surface contact point, ΔSCP represents the displacementbetween SCP_(CURRENT) 120 and the initial proposed surface contact pointSCP_(PROPOSED) 122, d represents the displacement between the currentsurface contact point SCP_(CURRENT) 120 and the current haptic interfacepoint P′_(HI) 118, and k_(f) represents the coefficient of kineticfriction.

Once the haptic rendering application has updated the location of theproposed surface contact point, in step 136 the haptic renderingapplication calculates a friction force to be applied to the user inresponse to the updated location. In calculating the friction force, thehaptic rendering application uses the coefficient of kinetic frictionassociated with the location of the updated proposed current surfacecontact point SCP′_(PROPOSED).

FIG. 8 shows a flowchart illustrating the steps performed by oneembodiment of the method of the present invention for determining theinitial proposed surface contact points for algebraic objects containedwithin the haptic interactive representation. As discussed above, analgebraic object is an object that is defined by an algebraic equation.The proposed surface contact points calculated with the methodillustrated in FIG. 8 are utilized in the methods outlined in theflowcharts in FIGS. 3, 4, 5 and 7. The method illustrated by theflowchart in FIG. 8 is used for the first collision of each algebraicobject with the haptic interface. The method for determining theproposed surface contact point for an algebraic object once the objecthas collided with the haptic interface and a first surface contact pointhas been determined will be discussed in more detail below in thedescription of FIG. 9.

In step 140, the user initiates interaction with the haptic interactiverepresentation. As described above, the sensors of the haptic interfacesystem sense the position of the user in real space. The hapticrendering application utilizes the information obtained by the sensorsto determine the user's haptic interface location in haptic interactionspace. Once the haptic rendering application has determined the user'shaptic interface, in step 142 the haptic rendering applicationdetermines if the haptic interface is free of collision with a virtualobject. If the haptic interface collides with at least one virtualobject, the haptic rendering application proceeds to step 144 and waitsfor the haptic interface to be free of collision with any virtualobjects. In other words, before the haptic rendering applicationdetermines any proposed surface contact points, the haptic interfacemust be free of collision with any of the virtual objects in the hapticinteractive representation. The position of the user in real space mustchange in order for the haptic interface to become free of collisionwith the virtual objects.

If the haptic interface is free of collision with any of the virtualobjects upon the user initiating interaction with the haptic interactiverepresentation, in step 146 the haptic rendering application determinesif the location of the haptic interface has changed. If the location ofthe haptic interface has not changed, in step 148 the haptic renderingapplication waits for the location of the haptic interface to change. Ifthe location of the haptic interface has changed, in step 150 the hapticrendering application determines if the haptic interface has collidedwith a virtual object. To determine if the haptic interface has collidedwith an object, the haptic rendering application evaluates the function(S(P)=0) defining the surface of algebraic object using the pointdefining the location of the haptic interface point. If the functionS(P) is equal to zero at the location of the haptic interface, then thehaptic interface is located directly on the surface of the algebraicobject and the haptic interface is deemed to have collided with theobject. If the function S(P) is less than zero at the location of thehaptic interface, then the haptic interface is located within thesurface of the object and the haptic interface is deemed to havecollided with the algebraic object. If the function S(P) is greater thanzero at the location of the haptic interface, then the haptic interfaceis located outside the algebraic object and the haptic interface isdeemed not to have collided with the algebraic object. If the hapticinterface has not collided with a virtual object, the haptic renderingapplication returns to step 146 and waits for the haptic interface tochange its location again.

If the haptic interface has collided with at least one virtual object,in step 152 the haptic rendering application determines the gradient ofthe surface (S′(P)) of the collided object at the current hapticinterface location. Next, in step 154, the haptic rendering applicationplaces the proposed surface contact point at the current hapticinterface location. The object of the remaining next group of steps 156,158 and 160 is to move the proposed surface contact point closer to thesurface defined by S(P)=0. The method moves the proposed surface contactpoint in successive steps from the haptic interface location toward thesurface in the direction of the gradient of the surface S′(P) until thesize of the step taken toward the surface is less than a predeterminedamount. To calculate the size of the step to be taken, in step 156 thehaptic rendering application evaluates the surface function S(P) usingthe proposed surface contact point. The haptic rendering applicationthen calculates the vector (dP) representing the step to be taken fromthe current proposed surface contact point toward the surface of theobject according to equation (4) below.

$\begin{matrix}{{dP} = \frac{{S(P)} \cdot {S^{\prime}(P)}}{{S^{\prime}(P)} \cdot {S^{\prime}(P)}}} & (4)\end{matrix}$

-   -   1. In equation (4), S(P) represents the value of the surface        function at the location of the current proposed surface contact        point and S′(P) represents the gradient of the surface defined        by the equation S(P). Once the haptic rendering application has        determined the vector dP, it calculates the magnitude (|dP|) of        the vector dP. Next, in step 158 the haptic rendering        application determines if the magnitude |dP| of the step needed        to be taken toward the surface of the algebraic object is        greater than a predetermined step size. The predetermined step        size indicates the proposed surface contact point is acceptably        close to the virtual object.

If the magnitude |dP| of the step is not greater than the predeterminedamount, then the haptic rendering application proceeds to step 160 andcalculates the force to be applied to the user in response to thecurrent proposed surface contact point. Next, in step 162, the hapticrendering application waits for a change in the location of the hapticinterface to determine a new surface contact point. If the magnitude|dP| of the step is greater than the predetermined amount, the hapticrendering application proceeds to step 164 and updates the location ofthe current surface contact point to a new surface contact pointSCP_(NEW) according to equation (5) below.

SCP_(NEW)=SCP_(CURRENT) −dP  (5)

In equation (5), SCP_(NEW) represents the vector pointing from theorigin of the coordinate system defining the haptic interface space tothe new proposed surface contact point, SCP_(CURRENT) represents thevector pointing from the origin to the current proposed surface contactpoint and dP represents the step to be taken from the current surfacecontact point to the surface of the object.

After the location of the surface contact point has been updated, thehaptic rendering application returns to step 156 and repeats steps 156,158 and 164 until the magnitude of the step taken toward the surface ofthe object is less than the predetermined amount.

In another embodiment, in step 158, rather than determining whether themagnitude of the step taken towards the surface of the object is greaterthan a predetermined amount, the haptic rendering application determinesif the result of evaluating the surface function equation S(P) in step156 for the current surface contact point is less than a predeterminedamount. Evaluating the surface function equation S(P) provides a resultthat is a measure of the distance of the proposed surface contact pointto the surface of the algebraic object.

FIG. 9 shows a flowchart illustrating the steps performed by oneembodiment of the method of the present invention for determining theproposed surface contact point for an algebraic object contained withinthe haptic interactive representation after the initial contact with thealgebraic object has been made and a first proposed surface contactpoint has been calculated. The haptic rendering application executes themethod of FIG. 9 after the haptic interface location changes in step 162of FIG. 8.

FIG. 9A shows an example of an algebraic surface 166 of an object 167defined by the function S(P) which has been generated by one embodimentof the present invention. FIG. 9A shows the haptic interface pointmoving from a current haptic interface point location (P_(HI)) 168 to asecond haptic interface point location (P′_(HI)) 170. The pointSCP_(CURRENT) 172 represents the current location of the surface contactpoint as calculated by the haptic rendering application upon the user'sfirst contact with the algebraic object 167. The haptic renderingapplication calculated the point SCP_(CURRENT) 172 in response to thefirst haptic interface point location (P_(HI)) 168. The pointSCP_(PROPOSED) 174 represents the haptic rendering application'sproposed new surface contact point. The haptic rendering applicationcalculated the point SCP_(PROPOSED) 174 in response to the change inlocation of the haptic interface point from P_(HI) 168 to P′_(HI) 170.

In step 164, the haptic rendering application determines the plane (T₁)176 tangent to the collided virtual object 167 at the current surfacecontact point location SCP_(CURRENT) 172. Next in step 178, the hapticrendering application determines the point (SCP_(TANGENT)) 180 on thetangent plane T₁ 176 that is closest to the new haptic interfacelocation P′_(HI) 170. The tangent plane T₁ 176 is defined by N_(T1) 182,a vector normal to the plane T₁ 176 and d_(TI), a scalar distance fromthe origin of the coordinate system defining the haptic interactionspace. To compute the location of SCP_(TANGENT) 180, the hapticrendering application determines the line 184 parallel to N_(T1) 182which passes through the point P′_(HI) 170. The haptic renderingapplication then determines the point of intersection 180 of the line184 with the plane T₁ 176. The point of intersection is the pointSCP_(TANGENT) 180. The haptic rendering application uses equation (6)below to calculate the location of SCP_(TANGENT) 180.

SCP_(TANGENT) =P′ _(HI)+(d _(TI) −P′ _(HI) ·N _(TI))N _(TI)  (6)

In equation (6), SCP_(TANGENT) represents the vector pointing from theorigin of the coordinate system to the point SCP_(TANGENT) 180.SCP_(TANGENT) 180 is now the new proposed surface contact pointlocation.

Next, the haptic rendering application determines the pointSCP_(PROPOSED) 174 on the surface 166 of the virtual object 167. Toiteratively move the point SCP_(TANGENT) 180 toward the surface, in step186 the haptic rendering application evaluates the surface function S(P)defining the surface 166 using the point SCP_(TANGENT) 180. In step 188,the haptic rendering application determines if the step taken fromSCP_(TANGENT) 180 to the surface 166 is greater than a predeterminedamount. If the step to be taken toward the surface is not greater than apredetermined amount, in step 190, the haptic rendering applicationplaces the current surface contact point at the current proposed surfacecontact point and calculates the force to be applied to the user in realspace in response to the new current surface contact point. Next, instep 192, the haptic rendering application waits for the hapticinterface location to change. Once the haptic interface locationchanges, the haptic rendering application returns to step 164 andrepeats the process to calculate the next surface contact pointlocation.

If the magnitude of the step to be taken toward the surface is greaterthan a predetermined amount, the haptic rendering application proceedsto step 194 and updates the proposed surface contact point location. Inone embodiment, the haptic rendering application uses a method similarto the method described above in steps 156, 158 and 164 of FIG. 8 toiteratively move the point SCP_(TANGENT) 180 closer to the pointSCP_(PROPOSED) 174.

FIG. 10A shows a flowchart illustrating the steps performed by oneembodiment of the method of the present invention for determining aproposed surface contact point for a polygonal object contained withinthe haptic interactive representation. As discussed above, a polygonalobject is an object created by linking polygons. Similar to the methodfor calculating the initial proposed surface contact point for algebraicobjects, after the user initiates interaction with the hapticinteractive representation, the haptic rendering application waits forthe haptic interface to be free of collision with any virtual object.Once the haptic interface is free of collision with any virtual object,the haptic rendering application then waits for the haptic interface tocollide with a virtual object. In one embodiment, to determine if thehaptic interface point collides with a virtual polygonal object, thehaptic rendering application determines a line intersecting the currenthaptic interface location and the previous haptic interface location. Ifthis line intersects any facets of a polygonal object, then the hapticinterface has collided with that object.

In step 200, the haptic rendering application sets the current surfacecontact point (SCP_(CURRENT)) equal to the last haptic interface point(P_(OUTSIDE)) not located within the polygonal object. Next, in step 202the haptic rendering application determines whether the currentlycollided object collided with the haptic interface location during theprevious iteration. If the virtual object did not collide with thehaptic interface locator during the previous iteration, then the hapticrendering application proceeds to step 204 and sets the proposed surfacecontact point (SCP_(PROPOSED)) equal to the current haptic interfacelocation (P_(HI)). Next, in step 206 the haptic rendering applicationdetermines the line segment L1 connecting P_(OUTSIDE) andSCP_(PROPOSED). The haptic rendering application then determines if theline segment L1 intersects any of the polygonal facets that compose thecollided polygonal object. If the line segment L1 intersects more thanone facet, the haptic rendering application determines the facet (Poly₁)that has a point of intersection that is closest to SCP_(CURRENT). Thehaptic rendering application repeats step 206 for all of the facets ofthe collided polygonal object.

Once the haptic rendering application has completed step 206 for all ofthe facets of the collided polygonal object, it proceeds to step 208 anddetermines if the line segment L1 intersected any of the polygonalobject's facets in step 206. If the line segment L1 did not intersectany of the polygonal facets in step 206, the polygonal object is notcolliding with the haptic interface. In step 210 the haptic renderinguses the proposed surface contact point SCP_(PROPOSED) as the proposedsurface contact point for that virtual object. If the line segment L1did intersect at least one of the polygonal facets in step 206, thepolygonal object is colliding with the haptic interface and the hapticrendering application proceeds to step 212 which will be discussed inmore detail below.

If in step 202 the haptic rendering application determines that thecurrently collided object collided with the haptic interface locationduring the previous iteration, the haptic rendering application proceedsto step 214. In step 214, the haptic rendering application determines aline segment L2 connecting the current surface contact pointSCP_(CURRENT) to the current haptic interface point P. Then the hapticrendering application determines if the line segment L2 intersects thepreviously intersected facet (POLY₁). If the line segment L2 does notintersect the previously intersected facet of the polygonal object, thenthe haptic rendering application proceeds to step 204 discussed above.If the line segment L2 intersects the previously intersected facetPOLY₁, then the haptic rendering application proceeds to step 212. Instep 212, the haptic rendering application determines the point(P_(POLYINTERSECT)) at which the line segment L2 intersects thepolygonal object's facet POLY₁. Next, the haptic rendering applicationprojects the current haptic interface point P_(HI) onto the planecontaining the facet POLY₁. The projected point is referred to herein asP_(PLANEPROJECT).

Next, in step 216, the haptic rendering application determines if thepoint P_(PLANEPROJECT) lies outside of any of the edges of facet POLY₁.To determine if the point P_(PLANEPROJECT) lies outside any of the edgesof facet POLY₁, the haptic rendering application determines a linesegment L3 which connects the point P_(POLYINTERSECT) to the pointP_(PLANEPROJECT). If the line segment L3 does not cross any of the edgesof POLY₁ then the point P_(PLANEPROJECT) lies within the facet POLY₁. Ifthe point P_(PLANEPROJECT) lies entirely within the boundaries of thefacet POLY₁, the haptic rendering application proceeds to step 218 andplaces the new proposed surface contact point at the pointP_(PLANEPROJECT). In another embodiment, the haptic renderingapplication places the new proposed surface contact point at a locationslightly above the surface of the facet POLY₁. The haptic renderingapplication calculates the point slightly above the facet POLY₁according to equation (7) below in which ε represents a predetermineddistance from the surface of the polygonal object, SCP PROPOSEDrepresents the vector pointing from the origin of the coordinate systemdefining the haptic interaction space to the proposed surface contactpoint, P_(PLANEPROJECT) represents the vector pointing from the originto point P_(PLANEPROJECT), and POLY_(1NORMAL) represents the unit vectornormal to the facet POLY₁.

SCP_(PROPOSED) =P _(PLANEPROJECT)+ε·POLY_(1NORMAL)  (7)

The value of ε is determined by how far above the surface the proposedsurface contact point should be located.

If the point P_(PLANEPROJECT) lies outside one of the boundaries of thefacet POLY₁, the haptic rendering application proceeds to step 220 inFIG. 10B. FIG. 10B shows a flowchart illustrating the steps performed byone embodiment of the method of the present invention for determining aproposed surface contact point for a polygonal object when the hapticinterface is not located solely within one facet of the polygonalobject. In step 220, the haptic rendering application determines whichedge (EDGE_(CROSSED)) of POLY₁ was crossed and the facets of thepolygonal object which share EDGE_(CROSSED). The haptic renderingapplication also determines which of the facets sharing EDGE_(CROSSED)are intersected by the line segment L3, which connects the pointP_(POLYINTERSECT) to the point P_(PLANEPROJECT). The haptic renderingapplication then selects one of the facets (POLY2) which sharesEDGE_(CROSSED). Next, in step 222 the haptic rendering applicationdetermines if the point P_(PLANEPROJECT) is below the facet POLY2. Thehaptic rendering application evaluates the equation defining the planecontaining the facet POLY2 using the point P_(PLANEPROJECT). If theresult of the equation is greater than zero, the point P_(PLANEPROJECT)is above the facet POLY2. If the result of the equation is less thanzero the point P_(PLANEPROJECT) is below the facet POLY2. If the pointP_(PLANEPROJECT) is above the facet POLY2, then the pointP_(PLANEPROJECT) is located outside the surface of the virtual polygonalobject and the haptic rendering application proceeds to step 224. Instep 224, the haptic rendering application determines the point(P_(EDGECROSSING)) where the line segment L3 intersects the edgeEDGE_(CROSSED) and places the proposed surface contact point at thepoint P_(EDGECROSSING).

In another embodiment, the haptic rendering application places the newproposed surface contact point at a location slightly above and to theside of the edge EDGE_(CROSSED). The haptic rendering applicationcalculates the point slightly above and to the side of the edgeEDGE_(CROSSED) according to equation (8) below in which ε represents apredetermined distance from the surface of the polygonal object,P_(JUST-BEYOND-EDGE) represents the vector pointing from the origin ofthe coordinate system defining the haptic interaction space to the newproposed surface contact point, P_(EDGECROSSING) represents the vectorpointing from the origin to point P_(EDGECROSSING), EDGE_(CROSSED)represents the vector along the edge EDGE_(CROSSED), and POLY_(1NORMAL)represents the unit vector normal to the facet POLY₁.

P _(JUSTBEYONDEDGE) =P_(EDGECROSSING)+εPOLY_(1NORMAL)+2·ε(POLY_(1NORMAL)×EDGE_(CROSSED))  (8)

If the point P_(PLANEPROJECT) is below the facet POLY2, then the pointP_(PLANEPROJECT) is located within the surface of the virtual polygonalobject and the haptic rendering application proceeds to step 226. Instep 226, the haptic rendering application projects the haptic interfacepoint P_(HI) along the edge EDGE_(CROSSED) to determine a pointP_(EDGEPROJECT). In step 228, the haptic rendering applicationdetermines if the point P_(EDGEPROJECT) is between the two vertices (endpoints) of the edge EDGE_(CROSSED). If the point P_(EDGEPROJECT) isbetween the two vertices, then the haptic rendering application proceedsto step 230 and places the new proposed surface contact point atP_(EDGEPROJECT). In another embodiment, the haptic rendering applicationplaces the new proposed surface contact point at a location slightlyabove the edge EDGE_(CROSSED) and inside the facet POLY₁ The hapticrendering application calculates the point slightly above the edgeEDGE_(CROSSED) according to equations (9) and (10) below in which εrepresents a predetermined distance from the surface of the polygonalobject, SCP_(PROPOSED) represents the vector pointing from the origin ofthe coordinate system defining the haptic interaction space to the newproposed surface contact point, P_(EDGEPROJECT) represents the vectorpointing from the origin to point P_(EDGEPROJECT), POLY_(1NORMAL)represents the unit vector normal to the facet POLY1, and POLY_(2NORMAL)represents the unit vector normal to the facet POLY₂.

SCP_(NORMAL)=normalize(POLY_(NORMAL)+POLY_(2NORMAL))  (9)

SCP_(PROPOSED) =P _(EDGEPROJECT)+εSCP_(NORMAL)  (10)

If the point P_(EDGEPROJECT) is not between the two vertices, then thehaptic rendering application proceeds to step 232 and determines thevertice (VERTICE_(CORNER)) of the edge EDGE_(CROSSED) thatP_(EDGEPROJECT) is closest to. Next, in step 238, the haptic renderingapplication places the proposed surface contact point at the pointVERTICE_(CORNER).

In another embodiment, the haptic rendering application does not placethe proposed surface contact point at the point VERTICE_(CORNER).Instead, the haptic rendering application determines whether the pointP_(EDGEPROJECT) is behind any of the other facets of the polygonalobject other than POLY1 and POLY2 that meet at the verticeVERTICE_(CORNER). If the point is not behind any other facets, thehaptic rendering application places the new proposed surface contactpoint at a point (P_(JUSTBEYONDVERTICE)) slightly above the pointVERTICE_(CORNER). The haptic rendering application calculates the pointslightly above the point VERTICE_(CORNER) according to equation (11)below in which ε represents a predetermined distance from the surface ofthe polygonal object, P_(JUSTBEYONDVERTICE) represents the vectorpointing from the origin of the coordinate system defining the hapticinteraction space to the new proposed surface contact point,P_(EDGEPROJECT) represents the vector pointing from the origin to pointP_(EDGEPROJECT), VERTICE_(CORNER) represents the vector normal to thesurface at the point VERTICE_(CORNER), POLY_(1NORMAL) represents theunit vector normal to the facet POLY1, and POLY_(2NORMAL) represents theunit vector normal to the facet POLY₂.

P_(JUSTBEYONDVERTICE)=VERTICE_(CORNER)+ε·normalize(POLY_(1NORMAL)+POLY_(2NORMAL))+2ε·normalize(P_(EDGEPROJECT)−VERTICE_(CORNER))  (11)

If the point is behind another facet, it is located within a corner andthe haptic rendering application places the new proposed surface contactpoint at a point slightly above the point VERTICE_(CORNER). However, inthis case, the haptic rendering application calculates the pointslightly above the point VERTICE_(CORNER) according to equation (12)below in which ε represents a predetermined distance from the surface ofthe polygonal object, P_(SCP) represents the vector pointing from theorigin of the coordinate system defining the haptic interaction space tothe new proposed surface contact point, VERTICE_(CORNER) represents thevector normal to the surface at the point VERTICE_(CORNER), andSCP_(aNORMAL) represents the average normal of the surface normals ofall the facets sharing VERTICE_(CORNER).

P _(SCP)=VERTICE_(CORNER)+εSCP_(aNORMAL)  (12)

FIG. 11 shows a flowchart illustrating the steps performed by anotherembodiment of the method of the present invention for determining theforces to be applied to a user through a haptic interface device. In theembodiment illustrated by the flowchart of FIG. 11, the user'sinteractions with the haptic interactive representation are reduced tothose of a point interacting with three dimensional objects in a threedimensional virtual environment. In other embodiments, the user'sinteractions are not reduced to those of a point interacting with threedimensional virtual objects. In other embodiments, the haptic interfacemay be a set of points. In still other embodiments, the haptic interfacemay be a three-dimensional object.

In step 250, the haptic rendering application generates a hapticinteractive representation which includes at least two representationsof virtual objects. FIG. 12A shows an example of a haptic interactiverepresentation created by one embodiment of the present invention whichcontains two virtual objects 252, 254 which do not overlap. FIG. 12Bshows an example of a haptic interactive representation created by oneembodiment of the present invention which contains two virtual objects252′, 254′ which overlap. Once the haptic rendering application hasgenerated a haptic interactive representation which includes at leasttwo virtual objects, in step 256 the haptic interface device senses theposition of the user in real space. In step 258, the haptic renderingapplication uses the position of the user in real space to determine thelocation of the haptic interface in the haptic interaction space. InFIGS. 12A and 12B the points PHI 260 and P′HI 260′ represent the user'shaptic interface point in the haptic interaction space. When the userchanges position, the haptic interface device senses this change inposition and the haptic rendering application updates the location ofthe haptic interface in haptic interaction space to reflect the changeof the user's position in real space.

Once the haptic rendering application determines the location of thehaptic interface, in step 262 it determines whether any of the virtualobjects collide with the haptic interface location. A method fordetermining whether a virtual object collides with the haptic interfacelocation will be discussed in more detail below. If no virtual objectscollide with the haptic interface, in step 264 the haptic renderingapplication calculates the force to be applied to the user in real spacebased on the location of the haptic interface. If at least one of thevirtual objects collides with the haptic interface location, the hapticrendering application proceeds to step 266. The objects which collidewith the haptic interface are termed current collision objects. In FIG.12A, object 254 is a current collision object and in FIG. 12B, objects252′ and 254′ are current collision objects.

In step 266, the haptic rendering application assigns one of the currentcollision objects to be the master collision object (MCO). In FIG. 12A,the object 254 is the MCO because there is only one current collisionobject. In FIG. 12B, the haptic rendering application assigned theobject 254′ to be the MCO. Next, in step 268, the haptic renderingapplication calculates a proposed surface contact point for the MCO. InFIG. 12A, the point SCP_(PROPOSED) 270 represents the proposed surfacecontact point for the MCO 254 and in FIG. 12B, the point SCP′_(PROPOSED)270′ represents the proposed surface contact point for the MCO 254′.Once the haptic rendering application has determined a proposed surfacecontact point for the MCO, in step 272 it determines a line segmentbetween the proposed surface contact point and the current surfacecontact point. In step 274, the haptic rendering application determinesif any virtual objects in the haptic interaction space other than theMCO intersect this line segment. For example, in FIG. 12A, the hapticrendering application determines the line 276 which intersects the pointSCP_(PROPOSED) 270 and the current surface contact point SCP_(CURRENT)278. Since no objects other than the MCO 254 intersect the line segment276, the haptic rendering application proceeds to step 280 and placesthe current surface contact point at the location of the MCO's proposedsurface contact point, SCP_(PROPOSED) 270. Next, in step 282 the hapticrendering application calculates the force to be applied to the user inreal space in response to the new current surface contact point.

If another virtual object intersect the line segment between the MCO'sproposed surface contact point and the current surface contact point,the haptic rendering application proceeds to step 284. For example, inFIG. 12B the line segment 276′ between the points SCP′_(PROPOSED) 270′and SCP′_(CURRENT) 278′ intersects the virtual object 252′. In step 284,the haptic rendering applicant determines the collision point(P_(COLLISION)) 286 between the line segment 276′ and the collisionobject 252′. Next, in step 288 the haptic rendering applicationdetermines the surface normal P _(COLLISION) 290 of the collision object252′ at the collision point P_(COLLISION) 286. In step 292 the hapticrendering application determines the surface normal SCP_(PROPOSED) 294at the MCO's proposed surface contact point SCP′_(PROPOSED) 270′. Next,in step 296 the haptic rendering application calculates the crossproducts of the vectors P _(COLLISION) 290 and SCP_(PROPOSED) 294. Next,in step 298 the haptic rendering application determines the line passingthrough the point P_(COLLISION) 286 and parallel to the vector resultingfrom the cross product of the vectors P _(COLLISION) 290 andSCP_(PROPOSED) 294. Next in step 300, the haptic rendering applicationprojects the MCO's proposed surface contact point SCP′_(PROPOSED) 270′onto the line determined in step 298 to determine an updated proposedsurface contact point location (SCP″_(PROPOSED)). After the hapticrendering application has determined the updated proposed surfacecontact point SCP″_(PROPOSED), it proceeds to step 302 in FIG. 11A.

In step 302, the haptic rendering application determines if the updatedproposed surface contact point SCP″_(PROPOSED), collides with any othervirtual objects, other than the objects 252′ and 254′. If the pointSCP″_(PROPOSED) does not collide with any other virtual objects, thehaptic rendering application proceeds to step 304 and places the currentsurface contact point at the location of the updated proposed surfacecontact point SCP″_(PROPOSED). Finally in step 306 the haptic renderingapplication calculates the force to be applied to the user in real spacein response t the new current surface contact point.

If the point SCP″_(PROPOSED) does collide with any other virtualobjects, the haptic rendering application proceeds to step 308 andplaces the current surface contact point at the intersection.

The computational independence of the haptic space from the otherspaces, such as graphics space and acoustic space which make up andinteract with the virtual environment permits the computationalmanipulation of each of the spaces to occur at a rate which is optimalfor the particular space. For example, although a user's hapticinteraction with the virtual environment may be sampled, processed andfed back to the user at a rate of 1000 times per second, the graphicsdisplay depicting the interaction needs to be refreshed only at a rateof fifty times per second. Therefore only one in every twenty samples ofwhat is occurring in haptic space need be depicted in graphics space.Similarly, the communication between, for example, haptic processes overa network also occurs at a rate slower than the rate at which the hapticspace is updated. Therefore, a mechanism to permit the synchronizationof processes executing at different rates is used to make use of thecomputational independence of the various processes manipulating andinteracting with a virtual environment.

Other advantages of computational independence include the use ofdifferent representations for haptic and graphic rending—such thatrepresentations best suited to each particular modality, for a givenapplication, may be used. In addition, the use of dissimilarrepresentations for the graphics and haptics processes, allowscompletely artificial worlds to be created where haptic and graphicalstimuli do not correspond.

One embodiment of a system using such synchronization is shown in FIG.13. In this system, a haptic process 1300 manipulating and interactingwith haptic space 1320 interacts with other processes such as a graphicsprocess 1340 and a communications process 1360 using a “callback”mechanism 1400. A callback 140 is a data structure which is associatedwith a virtual object by a process 1300 which created it. The datastructure contains the parameters necessary for an object in one spaceto be manipulated in a second space.

In the embodiment of the system as shown, each callback 1400, 1400′,1400″ (generally 1400) is associated by the haptic process 1300 whichcreated the callback 1400 with an object in haptic space 1320. Forexample, the data structure associated with the object in haptic spacewould include its X-, Y- and Z-coordinates in haptic space. A process,such as the graphics process 1340, wishing to communicate with thehaptic space 1320 in order to update the graphics space 1420, alsoassociates the callback 1400 with same object in graphics space 1420.Thus when the graphics process 1340 wishes to know the new location ofthe object for its depiction in graphics space 1420, the graphicsprocess 1340 accesses the callback 1400 associated with the object andthe haptic process 1300 supplies the data structure with the latest X-,Y- and Z-coordinate information regarding the object. The graphicsprocess 1340 then accesses the callback 1400 to retrieve the coordinateinformation and after making the appropriate transformations displaysthe new object location in graphics space 1420.

Similarly a second haptic process 1460 manipulating and interacting witha second haptic space 1480 can interact with the first haptic process1300 through a communications process 1360. In this case thecommunications process 1360 interacts with each haptic process 1300,1460 through a callback 1400′, 1400″ respectively associated with eachprocess. Thus when one haptic process 1300 needs information regardingthe forces on an object generated by a user using another haptic process1460, the communications process 1360 requests information from thesecond process 1460 through the callback 1400″. The haptic processrequest information from a second process through the same callback withwhich the second process requests data from the haptic process.

The steps therefore involved in using a callback are shown in FIG. 14. Ahaptic process is first created (step 1500) which defines objects in thehaptic space. The haptic process then creates a callback data structureand associates the data structure with an object in the haptic space(step 1520). A connection is then made with a second process, such as agraphics process (step 1540) and the callback is associated with anobject in the space of the second process (step 1560).

When the second process needs information about the object, the secondprocess makes a request of the callback (step 1580) and the hapticprocess is informed of the request (step 1600). The haptic process thenprovides the haptic parameters associated with the object to thecallback (step 1620). The second process then accesses the parameters inthe callback (step 1640) and uses these parameters to manipulate theobject in its space.

The capability of two haptic processes to communicate over acommunications link permits a user on one network node to experience thetactile sensation from a process operating on a second network node.This permits haptic virtual reality environments to be shared across thenetwork as easily as graphic virtual reality environments are presentlyshared.

Many typical virtual reality applications provide an image of an objecton a display with which a user can interact. Each of these images, orvirtual objects, are composed of many points. The complexity of thevirtual object determines how many points are needed to define theobject. Very complex objects require thousands of points in order forthe object to seem real to the user. In order for the virtual objects toseem “real” to the user, all the points defining the surface of theobject must be presented to the user at the same time. When the userinteracts with the object, all of the points composing the object mustbe updated to display the result of the interaction to the user. Forexample, if a user moves a virtual cube from a first location to asecond location, each point of the cube must be transformed to appear atthe second location on the display.

In contrast, in one embodiment of the present invention, only one pointof a haptic interactive representation is presented to a user at a time.A user is only able to “touch” and interact with one point of the hapticinteractive representation at a time. In one embodiment, to determinethe reaction forces to be applied to the user, the haptic renderingapplication transforms every point of each virtual object to determinethe corresponding points in real space and then calculates the reactionforces to be applied to a user in real space. In another embodiment, thehaptic rendering application transforms the position of the user in realspace into each virtual objects' space in the haptic interaction space.As used herein, an “object's space” refers to the coordinate systemdefining a particular object. Each object has its own coordinate systemwhich defines and travels with that object. The advantage of this methodis that the haptic rendering application only has to transform a singlepoint from real space into each of the object spaces rather thantransforming each of the points defining each of the objects in theobject spaces into real space. In the majority of cases, utilizing thismethod significantly decreases the amount of computations necessary tocalculate the appropriate reaction forces to be applied to the user inreal space. This method cannot be used by graphics applications becausegraphics applications present many points to a user at a time.

FIG. 15 shows a flowchart illustrating the steps performed by thisembodiment to determine the reaction forces to be applied to the user inreal space through a haptic interface device. In step 320, the hapticinterface device senses the position of the user in real space. Theuser's position in real space is represented by a set of real worldcoordinates. Next, in step 322, the haptic rendering applicationdetermines a first transform to transform the position of the user inreal space into a position in an object's space. In one embodiment, thefirst transform is a transformation matrix which transforms the user'sreal space coordinates into coordinates in the object's space. Thehaptic rendering application determines a transform for each object inthe haptic interaction space. In step 324 the haptic renderingapplication applies the transform to the user's real space coordinatesto obtain the user's coordinates in the object's space. In step 326, thehaptic rendering application determines if the user's coordinates in theobject's space collide with the virtual object. Also in step 326 thehaptic rendering application calculates the resulting reaction forces tobe applied to the user. The resulting reaction forces are represented byvectors. In step 328, the haptic rendering application determines asecond transform to transform the reaction force vectors calculated instep 326 from the object's space into a vector in real space. In oneembodiment, the second transform is a transformation matrix whichtransforms the vector in object space coordinates in to a vector in realspace coordinates. The haptic rendering application determines atransform for each object in the haptic interaction space. In step 330,the haptic rendering application applies the second transform to thereaction force vector in the object's space that was calculated in step326. Finally in step 332, the transformed reaction force vector isapplied to the user in real space.

In another embodiment, the haptic rendering application transforms a setof points representing the user' position in real space into eachobject's space.

The present invention also relates to an apparatus for generating andinterfacing with a haptic virtual reality environment. FIG. 16 shows anembodiment of an apparatus for enabling a user to generate and interfacewith a haptic virtual reality environment. The apparatus includes ahaptic interface device 350, a haptic rendering processor 352, a displayprocessor 354 and a display 356. The haptic interface device 350includes a sensor for sensing the position of the user 358 in real spaceand a force actuator for applying forces to the user in real space.

The haptic rendering processor 352 is in electrical communication withthe haptic interface device 350. The haptic rendering processor 352executes an algorithm to generate a haptic interactive representationand determine the forces to be applied to the user 358 in real space. Inone embodiment, the algorithm includes a module generating a hapticinteractive representation, a module determining the user's hapticinterface location in haptic interaction space, and a module calculatinga force to be applied to the user in real space. The module determiningthe user's haptic interface in haptic interaction space translates theposition of the user in real space into a position in haptic interactionspace. The module calculating a force to be applied to the userdetermines whether the user's haptic interface location collides withany of the virtual objects. If the user's haptic interface location doesnot collide with any virtual objects, then this module calculates aforce to be applied to the user in response to the user's hapticinterface location. If the user's haptic interface location does collidewith at least one virtual object then this module calculates a force tobe applied to the user in response to proposed surface contact pointsfor each virtual object. The haptic interface device 350 produces theforce calculated by the haptic rendering processor 352 and applies thecalculated force to the user 358.

The display processor 354 is in electrical communication with the hapticrendering processor 352. The display processor 354 displays therepresentation of the virtual object created by the haptic renderingprocessor on a display 356. The display 356 may be a computer screen,television screen, or any other device known in the art for displayingimages of objects. In one embodiment, the apparatus also includes anaudio processor 360 and a speaker 362. The audio processor 360 is inelectrical communication with the haptic rendering processor 352. Theaudio processor produces audio sounds in response to the interactions ofthe virtual objects and the user in the virtual environment. The speaker362 is in electrical communication with the audio processor 360. Thespeaker 362 outputs the sounds produced by the audio processor.

In one embodiment, the haptic rendering processor 352, the displayprocessor 354 and the audio processor 360 are different processors. Inanother embodiment, the haptic rendering processor 352, the displayprocessor 354 and the audio processor 360 are the same processor. In yetanother embodiment, the module generating a haptic interactiverepresentation, the module determining the user's haptic interfacelocation in haptic interaction space, and the module calculating a forceto be applied to the user in real space are separate devices.

In one embodiment, the apparatus shown in FIG. 16 for generating andinterfacing with a haptic virtual reality environment is a real-time,object-oriented, virtual environment creation tool which allows the user358 to create, modify, save and restore three-dimensional multi-modalvirtual environments using visual and haptic interaction techniques. Theresulting virtual environment can be felt, seen and heard by the user358.

FIG. 17 shows a flowchart illustrating the steps a user would perform tocreate and interface with a virtual environment using an embodiment ofthe apparatus described above and shown in FIG. 16. In step 370, theuser initiates the toolkit application. In one embodiment the displayprocessor 354 displays a graphical user interface on the display 356. Inone embodiment, the graphical user interface contains icons whichperforms functions when selected by the user 358. The icons may consistof geometric representations the user can select to build virtualobjects in the virtual environment. The icons may represent functionsfor operating on the geometric representations. The icons may alsoinclude different parameters that may be associated with a geometricrepresentation. These parameters may include textures, mass, velocity,acceleration, angular velocity, angular acceleration, inertia, friction,viscosity, compliance and other behavior parameters which may beassigned to the surfaces of virtual objects. In another embodiment, thegraphical display contains pull-down menus from which the user mayselect functions to be performed by the application.

To create the haptic interactive representation, in step 372 the userselects a base geometric representation for the virtual object that theuser desires to create. To select the object, the user manipulates thehaptic interface device 350 until the display 356 displays an image ofthe user's haptic interface location over the icon representing thegeometric representation. The user then elects the geometricrepresentation by “pressing” the icon. When the user presses the icon,the user experiences a force fed back through the haptic interfacedevice 350 which simulates the feel of pressing a button. Selecting thegeometric representation icon sends a signal to the haptic renderingprocessor 352 to create a virtual object. The haptic rendering processor352 then places a virtual object in the shape of the selected geometricrepresentation into the haptic interactive representation. The user maynow haptically interact with the virtual object through the hapticinterface device 350. The display processor 354 also places an image ofthe virtual object on the display 356. If the user desires, in step 374the user may translate the virtual object anywhere within the hapticinteraction space. To translate the object, the user selects thetranslate command and uses the haptic interface device 350 to come intocontact with the object. Once the user is in contact with the object,the user may “pick up” the object and place the object in the desiredlocation. Similarly, in step 376, the user may rotate the virtualobject. In step 578, the user may scale the virtual object to thedesired shape and size by selecting the scale command and contacting apoint on the surface of the virtual object. Once the user has contacteda point, the user may “grab” that point and stretch the virtual objectto the desired size and shape. In step 380, the user may assign anyother desired properties to the object. These properties may includemass, velocity, acceleration, texture, or any of the other surface anddynamic properties described above. The user may then repeat thisprocess to place another virtual object in the haptic interactiverepresentation.

In another embodiment, the user may manipulate the visual appearance ofthe virtual objects on the display 356. In yet another embodiment, theuser may assign audio sounds to the interactions in the virtualenvironment.

In one embodiment, the user may save the haptic interactiverepresentation and recall it later. Upon recall, the user may interactwith the virtual objects in the haptic interactive representation.

Having described preferred embodiments of the invention, it will nowbecome apparent to one of skill in the art that other embodimentsincorporating the concepts may be used. It is felt, therefore, thatthese embodiments should not be limited to disclosed embodiments butrather should be limited only by the spirit and scope of the followingclaims.

1-25. (canceled)
 26. A method for generating a haptic interactiverepresentation, comprising: defining a haptic interaction space;rendering a plurality of virtual objects in said haptic interactionspace, wherein said plurality of virtual objects comprises at least twomembers selected from the group consisting of a primitive object, apolygonal object, an algebraic object, a constructed solid geometryobject, an algebraic surface representation, and a volumetric objectrepresentation; and delivering force feedback to a user through a hapticinterface device according to an interaction in said haptic interactionspace.
 27. The method of claim 26, wherein at least one of saidplurality of virtual objects is a voxel representation.
 28. The methodof claim 26, wherein the rendering step comprises storing said pluralityof virtual objects.
 29. The method of claim 26, wherein the renderingstep comprises displaying said plurality of virtual objects.